Fisher Information Matrix Research Articles

Modelling Wind Speed, Humidity, and Temperature in Butterworth and Melaka during Southwest Monsoon in 2020 with a Simultaneous Linear Functional Relationship

The extension of parameter estimation from a bivariate linear functional relationship model (LFRM) to simultaneous LFRM for linear variables using the maximum likelihood estimation (MLE) method is explored in this paper. The covariance matrix of the parameter estimates is derived through the Fisher information matrix. A simulation study was done to investigate the performance of the parameter estimation. According to the simulation study, the estimated parameters have a small bias. The beauty of simultaneous LFRM lies in developing the model to study the relationship between more than two linear variables while considering error terms for all variables. The applicability of the proposed simultaneous model is demonstrated using wind speed, humidity, and temperature data from Butterworth and Melaka during the southwest monsoon season of 2020.

Bayesian design of clinical trials using the scale transformed power prior

ABSTRACT There are many Bayesian design methods allowing for the incorporation of historical data for sample size determination (SSD) in situations where the outcome in the historical data is the same as the outcome of a new study. However, there is a dearth of methods supporting the incorporation of data from a previously completed clinical trial that investigated the same or similar treatment as the new trial but had a primary outcome that is different. We propose a simulation-based Bayesian SSD framework using the partial-borrowing scale transformed power prior (straPP). The partial-borrowing straPP is developed by applying a novel scale transformation to a traditional power prior on the parameters from the historical data model to make the information better align with the new data model. The scale transformation is based on the assumption that the standardized parameters (i.e., parameters multiplied by the square roots of their respective Fisher information matrices) are equal. To illustrate the method, we present results from simulation studies that use real data from a previously completed clinical trial to design a new clinical trial with a primary time-to-event endpoint.

Layout of Detection Array Based on Multi-Strategy Fusion Improved Adaptive Mayfly Algorithm in Bearing-Only Sensor Network.

The various applications of bearing-only sensor networks for detection and localization are becoming increasingly widespread and important. The array layout of the bearing-only sensor network seriously impacts the detection performance. This paper proposes a multi-strategy fusion improved adaptive mayfly algorithm (MIAMA) in a bearing-only sensor network to perform layout planning on the geometric configuration of the optimal detection. Firstly, the system model of a bearing-only sensor network was constructed, and the observability of the system was analyzed based on the Cramer-Rao Lower Bound and Fisher Information Matrix. Then, in view of the limitations of the traditional mayfly algorithm, which has a single initial population and no adaptability and poor global search capabilities, multi-strategy fusion improvements were carried out by introducing Tent chaos mapping, the adaptive inertia weight factor, and Random Opposition-based Learning. Finally, three simulation experiments were conducted. Through comparison with the Particle Swarm Optimization (PSO) algorithm, Mayfly Algorithm (MA), and Genetic Algorithm (GA), the effectiveness and superiority of the proposed MIAMA were validated.

Optimizing Variational Quantum Algorithms with qBang: Efficiently Interweaving Metric and Momentum to Navigate Flat Energy Landscapes

Variational quantum algorithms (VQAs) represent a promising approach to utilizing current quantum computing infrastructures. VQAs are based on a parameterized quantum circuit optimized in a closed loop via a classical algorithm. This hybrid approach reduces the quantum processing unit load but comes at the cost of a classical optimization that can feature a flat energy landscape. Existing optimization techniques, including either imaginary time-propagation, natural gradient, or momentum-based approaches, are promising candidates but place either a significant burden on the quantum device or suffer frequently from slow convergence. In this work, we propose the quantum Broyden adaptive natural gradient (qBang) approach, a novel optimizer that aims to distill the best aspects of existing approaches. By employing the Broyden approach to approximate updates in the Fisher information matrix and combining it with a momentum-based algorithm, qBang reduces quantum-resource requirements while performing better than more resource-demanding alternatives. Benchmarks for the barren plateau, quantum chemistry, and the max-cut problem demonstrate an overall stable performance with a clear improvement over existing techniques in the case of flat (but not exponentially flat) optimization landscapes. qBang introduces a new development strategy for gradient-based VQAs with a plethora of possible improvements.

Objective Bayesian estimation for multistate stress‐strength model's reliability with various kernel functions

AbstractThe paper discusses the objective Bayesian estimation of the reliability of a multistate stress‐strength model (MSSM) based on different kernel functions. For the MSSM, we first derive the reliability function and Fisher information matrix. The Jeffreys prior, reference prior, and probability matching prior for the reliability function of the MSSM are constructed based on the objective Bayesian paradigm. Subsequently, we demonstrated that these priors are improper density, then evaluated the effects of these priors on Bayes estimates for MSSM's reliability based on a complete sample. The Bayesian estimates are calculated using random walk Metropolis–Hastings techniques. We employ Monte Carlo simulation to examine the effectiveness of Bayes estimates for MSSM's reliability in terms of average bias and mean squared error, meanwhile the highest posterior density credible intervals are investigated in terms of average length and coverage probability. Finally, two real datasets were examined, demonstrating the viability of the objective Bayes technique for small sample data.

On the entropy and information of Gaussian mixtures

AbstractWe establish several convexity properties for the entropy and Fisher information of mixtures of centred Gaussian distributions. Firstly, we prove that if are independent scalar Gaussian mixtures, then the entropy of is concave in , thus confirming a conjecture of Ball, Nayar and Tkocz (2016) for this class of random variables. In fact, we prove a generalisation of this assertion which also strengthens a result of Eskenazis, Nayar and Tkocz (2018). For the Fisher information, we extend a convexity result of Bobkov (2022) by showing that the Fisher information matrix is operator convex as a matrix‐valued function acting on densities of mixtures in . As an application, we establish rates for the convergence of the Fisher information matrix of the sum of weighted i.i.d. Gaussian mixtures in the operator norm along the central limit theorem under mild moment assumptions.

RRL: Recommendation Reverse Learning

As societies become increasingly aware of data privacy, regulations require that private information about users must be removed from both database and ML models, which is more colloquially called `the right to be forgotten`. Such privacy problems of recommendation systems, which hold large amounts of private data, are drawing increasing attention. Recent research suggests dividing the preference data into multiple shards and training submodels with these shards and forgetting users' personal preference data by retraining the submodels of marked shards. Despite the computational efficiency development compared with retraining from scratch, the overall recommendation performance deteriorates after dividing the shards because the collaborative information contained in the training data is broken. In this paper, we aim to propose a forgetting framework for recommendation models that neither separate the training data nor jeopardizes the recommendation performance, named Recommendation Reverse Learning (RRL). Given the trained recommendation model and marked preference data, we devise Reverse BPR Objective (RBPR Objective) to fine-tune the recommendation model to force it to forget the marked data. Nevertheless, as the recommendation model encode the complex collaborative information among users, we propose to utilize Fisher Information Matrix (FIM) to estimate the influence of reverse learning on other users' collaborative information and guide the updates of representations. We conduct experiments on two representative recommendation models and three public benchmark datasets to verify the efficiency of RRL. To verify the forgetting completeness, we use RRL to make the recommendation model poisoned by shilling attacks forget malicious users.

Fast Machine Unlearning without Retraining through Selective Synaptic Dampening

Machine unlearning, the ability for a machine learning model to forget, is becoming increasingly important to comply with data privacy regulations, as well as to remove harmful, manipulated, or outdated information. The key challenge lies in forgetting specific information while protecting model performance on the remaining data. While current state-of-the-art methods perform well, they typically require some level of retraining over the retained data, in order to protect or restore model performance. This adds computational overhead and mandates that the training data remain available and accessible, which may not be feasible. In contrast, other methods employ a retrain-free paradigm, however, these approaches are prohibitively computationally expensive and do not perform on par with their retrain-based counterparts. We present Selective Synaptic Dampening (SSD), a novel two-step, post hoc, retrain-free approach to machine unlearning which is fast, performant, and does not require long-term storage of the training data. First, SSD uses the Fisher information matrix of the training and forgetting data to select parameters that are disproportionately important to the forget set. Second, SSD induces forgetting by dampening these parameters proportional to their relative importance to the forget set with respect to the wider training data. We evaluate our method against several existing unlearning methods in a range of experiments using ResNet18 and Vision Transformer. Results show that the performance of SSD is competitive with retrain-based post hoc methods, demonstrating the viability of retrain-free post hoc unlearning approaches.

Resource Optimization Using a Step-by-Step Scheme in Wireless UWB Sensing and Localization Networks

Wireless localization and target sensing both rely on precise extraction of parameters such as signal amplitude, propagation delay, and Doppler shift from the received signals. Due to the high multi-path resolution and strong penetration, both localization and sensing can be achieved through identical UWB waveforms. In this paper, we try to properly allocate resources for localization and sensing to fully exploit the potential of UWB systems. Considering the complexity of the multi-slot networks, we derive the Fisher Information Matrix (FIM) expressions for single-slot and dual-slot integrated sensing and localization (ISAL) networks, respectively, and propose two resource optimization schemes, namely the step-by-stepscheme and the integrated scheme, respectively. The numerical results show that: (i) for the sensing-resource-limited networks with relatively uniform node distribution, the energy allocated to each step in the step-by-step scheme satisfies the relationship energy for clock offset < energy for radar localization < energy for target sensing; (ii) in the multi-slot ISAL networks, more energy will be allocated to the time slots where the networks are relatively sensing-resource-limited; (iii) the step-by-step scheme is more suitable for the sensing-resource-abundant networks, while the integrated scheme is more suitable for the sensing-resource-limited networks.

Multiple arbitrarily inflated Poisson regression analysis

In this paper, a new flexible count regression analysis is proposed. For this purpose, a new modification of the Poisson distribution is introduced which generalizes the Poisson, zero-inflated Poisson, zero-one inflated Poisson, and zero-one-two inflated Poisson distributions. Some distributional properties are discussed for the proposed distribution. The Fisher scoring and EM algorithms are derived to attain the maximum likelihood estimates of the unknown parameters. An expected Fisher information matrix is provided to construct an approximate confidence interval for the parameters. Using the modified Poisson distribution, an arbitrary multiply inflated counting regression model is proposed. The performance of the maximum likelihood methodology is investigated with a simulation study for the distribution and count regression model. Finally, two practical data sets are analyzed and the superiority of the proposed model is demonstrated among others.

Inference on the reliability of inverse Weibull with multiply Type-I censored data

Inverse Weibull (IW) distribution is widely used due to its non-monotonic hazard function. For the IW distribution, existing research on statistical inference has mostly focused on censored data, but there has been no study on multiply Type-I censoring until now as far as we know. The multiple Type-I censoring, acted as an extended form of Type-I censoring, is frequently encountered in industry and medicine research, which also needs to be paid attention. Thus, this study conducts detailed analyses on the reliability of IW distribution with multiply Type-I censored data, including point estimation and confidence interval (CI) construction. Four methods, including the maximum likelihood estimation (MLE), least square estimation (LSE), and two Bayesian estimations, namely MCMC and Lindley, are adopted in point estimation. Results show that the Lindley method performs best with small mission times, while MLE is optimal for large mission times. Specifically, for the CIs construction, we proposed a pivotal quantity based on LSE to construct the CIs of reliability and compare it with two popular methods, Fisher information matrix (FIM) derivation and MCMC algorithm. Our proposed method shows competitive performance as the MCMC and outperforms FIM. Finally, those estimation methods are applied to an example for illustration.

Statistical inference for a two-parameter Rayleigh distribution under generalized progressive hybrid censoring scheme

The two-parameter Rayleigh distribution, as an extended distribution of the Rayleigh distribution, has been widely applied in reliability analysis. With the introduction of the location parameter, two-parameter Rayleigh distribution becomes more flexible in fitting real-data. In this paper, based on generalized progressively hybrid censored(GPHC) sample from the two-parameter Rayleigh distribution, Classical and Bayesian inferences are discussed. The Newton-Raphson(NR) and Expectation-Maximization(EM) algorithms are used to compute the maximum likelihood estimates(MLEs). As well as the asymptotic confidence interval(ACI) estimation is obtained through the asymptotic distribution theory of maximum likelihood estimation(MLE) and computation of the observed Fisher information matrix. In Bayesian frame, the estimation of unknown parameters and prediction of future observable are taken into consideration. Due to Bayesian estimation is challenging to compute precisely and for the purpose of comparison, the Lindley's approximation, the Tierney-Kadane(TK) approximation and Markov chain Monte Carlo(MCMC) method are employed to obtain Bayesian estimates. Then, combining the MCMC algorithm mentioned in the article, the one- and two- samples Bayesian prediction are obtained. Finally, the simulation results are provided and a real-life data set is used for illustration purpose.

Inference for compound truncated Poisson log-normal model with application to maximum precipitation data

The main objective of this paper is to propose a new general family of distributions, namely compound truncated Poisson log-normal distribution of which log-normal distribution is a special case. The proposed model has three unknown parameters, and it can take variety of shapes. It can be used effectively in analyzing maximum precipitation data during a particular period of time obtained from different stations. It is assumed that the number of stations operate follows a zero-truncated Poisson random variables, and the daily precipitation follows a log-normal random variable. The maximum likelihood estimators can be obtained quite conveniently using Expectation-Maximization (EM) algorithm. Approximate maximum likelihood estimators are also derived. The associated confidence intervals can also be obtained from the observed Fisher information matrix. Simulation results have been performed to check the performance of the EM algorithm, and it is observed that the EM algorithm works quite well in this case. When we analyze the precipitation data set using the proposed model it is observed that the proposed model provides better fit than some of the existing models.

Обобщённое неравенство Крамера – Рао в случае совместного пеленгования по азимуту и углу места в условиях сложной электромагнитной обстановки

In this work, a generalized expression is obtained for the Cramer–Rao lower bound in the case of a joint estimation of azimuth and elevation angle in an arbitrary coordinate system in which the vector complex radiation pattern is determined, under conditions of a complex electromagnetic environment, including the presence of noise correlation in spatial channels and differences in its intensity. A theorem is formulated for the Fisher information of joint estimates of azimuth and elevation angle in the case of a multi-channel detector-direction finder with an antenna system described in the selected coordinate system by a vector complex radiation pattern. Dependences of the root-mean-square error of direction finding in azimuth and elevation on the signal-to-noise ratio and true directions to the radio source are obtained, calculated using known and generalized expressions for the Cramer–Rao lower bound. The condition for the applicability of the existing and proposed in the article expression for the Fisher information matrix is obtained. The concept of the “energy” center of the antenna system is given. To confirm the reliability of the obtained generalized expressions for the lower bound of the dispersion of the Cramer–Rao inequality, statistical modeling is carried out. The use of the obtained expressions in practice will allow the most optimal synthesis of antenna devices depending on the structure and characteristics of the direction finder antenna system, the number of antennas and the characteristics of the radio receiving path. Generalized expressions for the lower Cramer–Rao bound are the foundation for organizing the operation of intelligent radio monitoring systems, allowing for a well-founded choice of the optimal antenna array structure that provides the required direction-finding efficiency.

CL-BPUWM: continuous learning with Bayesian parameter updating and weight memory

AbstractCatastrophic forgetting in neural networks is a common problem, in which neural networks lose information from previous tasks after training on new tasks. Although adopting a regularization method that preferentially retains the parameters important to the previous task to avoid catastrophic forgetting has a positive effect; existing regularization methods cause the gradient to be near zero because the loss is at the local minimum. To solve this problem, we propose a new continuous learning method with Bayesian parameter updating and weight memory (CL-BPUWM). First, a parameter updating method based on the Bayes criterion is proposed to allow the neural network to gradually obtain new knowledge. The diagonal of the Fisher information matrix is then introduced to significantly minimize computation and increase parameter updating efficiency. Second, we suggest calculating the importance weight by observing how changes in each network parameter affect the model prediction output. In the process of model parameter updating, the Fisher information matrix and the sensitivity of the network are used as the quadratic penalty terms of the loss function. Finally, we apply dropout regularization to reduce model overfitting during training and to improve model generalizability. CL-BPUWM performs very well in continuous learning for classification tasks on CIFAR-100 dataset, CIFAR-10 dataset, and MNIST dataset. On CIFAR-100 dataset, it is 0.8%, 1.03% and 0.75% higher than the best performing regularization method (EWC) in three task partitions. On CIFAR-10 dataset, it is 2.25% higher than the regularization method (EWC) and 0.7% higher than the scaled method (GR). It is 0.66% higher than the regularization method (EWC) on the MNIST dataset. When the CL-BPUWM method was combined with the brain-inspired replay model under the CIFAR-100 and CIFAR-10 datasets, the classification accuracy was 2.35% and 5.38% higher than that of the baseline method, BI-R + SI.

Variational quantum algorithm for experimental photonic multiparameter estimation

Variational quantum metrology represents a powerful tool to optimize estimation strategies, resulting particularly beneficial for multiparameter estimation problems that often suffer from limitations due to the curse of dimensionality and computational complexity. To overcome these challenges, we develop a variational approach able to efficiently optimize a quantum multiphase sensor. Leveraging the reconfigurability of an integrated photonic device, we implement a hybrid quantum-classical feedback loop able to enhance the estimation performances. The quantum circuit evaluations are used to compute the system partial derivatives by applying the parameter-shift rule, and thus reconstruct experimentally the Fisher information matrix. This in turn is adopted as the cost function of a classical learning algorithm run to optimize the measurement settings. Our experimental results showcase significant improvements in estimation accuracy and noise robustness, highlighting the potential of variational techniques for practical applications in quantum sensing and more generally in quantum information processing using photonic circuits.

RIS-assisted near-field localization using practical phase shift model

Our research focuses on examining the problem of localizing user equipment (UE) in the uplink scenario using reconfigurable intelligent surfaces (RIS) based lens. We carry out a thorough analysis of the Fisher information matrix (FIM) and assess the influence of various RIS-based lens configurations using an actual RIS phase-dependent amplitude variations model. Furthermore, to reduce the complexity of the maximum likelihood (ML) estimator, a simple localization algorithm-based angular expansion is presented. Simulation results show superior localization performance when prior location information is available for directional and positional channel configurations. The position error bound (PEB) and the root mean square error (RMSE) are studied to evaluate the localization accuracy of the user utilizing the realistic RIS phase-dependent amplitude model in the near-field region. Furthermore, the achievable data rate is obtained in the same region using the realistic RIS phase-dependent amplitude model. It is noticed that adopting the actual RIS phase-dependent amplitude model under the near-field channel increases the localization error and degrades the data rate performance for amplitude value less than one so, the unity assumption of the RIS phase shift model used widely in the literature is inaccurate.

Continual online learning-based optimal tracking control of nonlinear strict-feedback systems: application to unmanned aerial vehicles

A novel optimal trajectory tracking scheme is introduced for nonlinear continuous-time systems in strict feedback form with uncertain dynamics by using neural networks (NNs). The method employs an actor-critic-based NN backstepping technique for minimizing a discounted value function along with an identifier to approximate unknown system dynamics that are expressed in augmented form. Novel online weight update laws for the actor and critic NNs are derived by using both the NN identifier and Hamilton-Jacobi-Bellman residual error. A new continual lifelong learning technique utilizing the Fisher Information Matrix via Hamilton-Jacobi-Bellman residual error is introduced to obtain the significance of weights in an online mode to overcome the issue of catastrophic forgetting for NNs, and closed-loop stability is analyzed and demonstrated. The effectiveness of the proposed method is shown in simulation by contrasting the proposed with a recent method from the literature on an underactuated unmanned aerial vehicle, covering both its translational and attitude dynamics.

Dynamic Weighted Cumulative Residual Entropy Estimators for Laplace Distribution: Bayesian Approach

Objectives: To develop Bayesian estimators of dynamic weighted cumulative residual entropy (DWCRE) for Laplace distribution and to investigate posterior risks using various priors and loss functions. Methods: Weighted entropy measure of information is provided by a probabilistic experiment whose basic events are described by their objective probabilities and some qualitative (objective or subjective) weights. In this paper, we have used priors (Jeffrey’s, Hartigan, Uniform and Gumble Type II) and several loss functions. Findings: Bayesian estimators and associated posterior risks for Laplace distribution have been derived for different priors and loss functions. Monte Carlo Simulation study and graphical analyses have also been presented along with the conclusion. Through the comprehensive simulation study in the paper, it has been observed that Hartigan prior is better than other priors in terms of the posterior risk whereas Uniform prior has always higher posterior risk. Novelty: The introduction of new Bayesian estimators and their posterior risks for dynamic weighted cumulative residual entropy (DWCRE) of Laplace distribution. Keywords: Bayesian estimators, Laplace distribution, Fisher information matrix, Loss functions, Priors

Quantum Fisher information matrix of quantum metrology in a Heisenberg XXZ model

Quantum Fisher information matrix of quantum metrology in a Heisenberg XXZ model